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ABSTRACT 

The detection and estimation of signals in noisy, limited 
data is a problem of interest to many scientific and engi- 
neering communities. We present a computationally sim- 
ple, sample eigenvalue based procedure for estimating 
the number of high-dimensional signals in white noise 
when there are relatively few samples. We highlight a 
fundamental asymptotic limit of sample eigenvalue based 
detection of weak high-dimensional signals from a lim- 
ited sample size and discuss its implication for the detec- 
tion of two closely spaced signals. 

This motivates our heuristic definition of the effec- 
tive number of identifiable signals. Numerical simula- 
tions are used to demonstrate the consistency of the algo- 
rithm with respect to the effective number of signals and 
the superior performance of the algorithm with respect 
to Wax and Kailath's "asymptotically consistent" MDL 
based estimator. 

Index Terms — Signal detection, eigen-inference, ran- 
dom matrices 

1. INTRODUCTION 

The observation vector, in many signal processing appli- 
cations, can be modelled as a superposition of a finite 
number of signals embedded in additive noise. Detect- 
ing the number of signals present becomes a key issue 
and is often the starting point for the signal parameter es- 
timation problem. When the signals and the noise are 
assumed to be samples of a stationary, ergodic Gaus- 
sian vector process, the sample covariance matrix formed 
from in observations has the Wishart distribution. 

The proposed algorithm uses an information theo- 
retic criterion, motivated by the approach taken by Wax 
and Kailath (henceforth WK) in (TJ, for determining the 
number of signals in white noise by performing inference 
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on the eigenvalues of the resulting sample covariance ma- 
trix. The form of the estimator is motivated by the dis- 
tributional properties of moments of the eigenvalues of 
large dimensional Wishart matrices Q. 

The proposed estimator was derived by explicitly ac- 
counting for the blurring and fluctuations of the eigenval- 
ues due to sample size constraints. Consequently, there is 
a greater theoretical justification for employing the pro- 
posed estimator in sample starved settings unlike the WK 
estimators which were derived assuming that the sam- 
ple size greatly exceeds the number of sensors. This 
is reflected in the improved performance relative to the 
"asymptotically consistent" WK MDL based estimator. 

Another important contribution of this paper is the 
description of a fundamental limit of eigen-inference, i.e., 
inference using the sample eigenvalues alone. The con- 
cept of effective number of identifiable signals, introduced 
herein, explains why, asymptotically, if the signal level 
is below a threshold that depends on the noise variance, 
sample size and the dimensionality of the system, then 
reliable detection is not possible. 

This paper is organized as follows. The problem is 
formulated in Section |2] An estimator for the number of 
signals present that exploits results from random matrix 
theory is derived in Section [3] The fundamental limits 
of sample eigenvalue based detection and the concept of 
effective number of signals are discussed in Section @] 
Simulation results are presented in Section [5] while some 
concluding remarks and directions for future research are 
presented in Section[6] 

2. PROBLEM FORMULATION 

We observe m samples ("snapshots") of possibly sig- 
nal bearing n-dimensional snapshot vectors xi , . . . , x m 
where for each i, Xj ~ Af n (0, R) and x, are mutually 
independent. The snapshot vectors are modelled as 
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x, = As, +z, for i = 1, . . . ,m, (1) 



where z; ~ JV„ (0, a 2 T), denotes an n-dimensional (real 
or complex) Gaussian noise vector where a 2 is generi- 
cally unknown, Sj ~ A4(0, R s ) denotes a fc-dimensional 
(real or complex) Gaussian signal vector with covariance 
R s , and A is a n x k unknown non-random matrix. 

Since the signal and noise vectors are independent 
of each other, the covariance matrix of can hence be 
decomposed as 

R = * + cr 2 I (2) 

where 

*=AR S A', (3) 

with ' denoting the conjugate transpose. Assuming that 
the matrix A is of full column rank, i.e., the columns of 
A are linearly independent, and that the covariance ma- 
trix of the signals R s is nonsingular, it follows that the 
rank of \& is k. Equivalently, the n — k smallest eigen- 
values of * are equal to zero. 

If we denote the eigenvalues of R by Ai > A2 > 
. . . > A n then it follows that the smallest n — k eigenval- 
ues of R are all equal to a 2 so that 

Afc+i = Afc+2 = • • • = A„ = A = a 2 . (4) 

Thus, if the true covariance matrix R were known apri- 
ori, the dimension of the signal vector k can be deter- 
mined from the multiplicity of the smallest eigenvalue 
of R. The problem in practice is that the covariance 
matrix R is unknown so that such a straight-forward al- 
gorithm cannot be used. The signal detection and es- 
timation problem is hence posed in terms of an infer- 
ence problem on m samples of n-dimensional multivari- 
ate real or complex Gaussian snapshot vectors. 

A classical approach to this problem, developed by 
Bartlett [3 | and Lawley J4], uses a sequence of hypothe- 
sis tests. Though this approach is sophisticated, the main 
problem is the subjective judgement needed by the prac- 
titioner in selecting the threshold levels for the differ- 
ent tests. This was overcome by Wax and Kailath in 
JT] wherein they propose an estimator for the number 
of signals (assuming m > n) based on the eigenvalues 
h > h > ■ ■ ■ > In of the sample covariance matrix 
(SCM) defined by 

1 m 1 
R = — V x t x' = —XX' (5) 
m z — ' m 

i=l 

where X = [xi| . . . |x TO ] is the matrix of observations 
(samples). The Akaike Information Criteria (AIC) form 
of the estimator is given by 

g(k) 

&AIC = argmin — 2(n — k)m log — - — + 2fc(2n — k) 

feeN:0<fc<n a{k) 

(6) 



while the Minimum Descriptive Length (MDL) criterion 
is given by 



1 r i\ 1 9\ K 

Kmdl= argmin -(n - fcjmlog — — ■ 

fcSN:0<fc<n a[K) 

+ -k(2n- fc)logm (7) 

where g(k) = n"=k+i ^ s ^ e geometric mean 

of the n — k smallest sample eigenvalues and a(k) = 
7T-k Ej=fc+i h i s me i r arithmetic mean. 

It is known |T| that the AIC form inconsistently es- 
timates the number of signals, while the MDL form es- 
timates the number of signals consistently. The simplic- 
ity of the estimator, and the large sample consistency are 
among the primary reasons why the Kailath-Wax MDL 
estimator continues to be employed in practice |5i]. In the 
two decades since the publication of the WK paper, re- 
searchers have come up with many innovative solutions 
(0 El El to list a few) for making the estimators more 
robust by exploiting some type of prior knowledge. 

The most important deficiency of the WK and re- 
lated estimators that remains unresolved occurs when the 
sample size is smaller than the number of sensors, i.e., 
when m < n. In this situation, the SCM is singular 
and the estimators become degenerate. Practitioners of- 
ten overcome this in an ad-hoc fashion by, for exam- 
ple, restricting A; in (O to integer values in the range 
< k < min(n, m). Since large sample, i.e., m ^> n, 
asymptotics were used to derive the estimators in (TJ, 
there is no rigorous theoretical justification for such a re- 
formulation even if the simulation results suggest that the 
WK estimators are working "well enough." 

Other sample eigenvalue based solutions found in the 
literature that exploit the sample eigenvalue order statis- 
tics |6), or employ a Bayesian framework by imposing 
priors on the number of signals (9) are computationally 
more intensive and do not address the sample size starved 
setting in their analysis or their simulations. Particle fil- 
ter based techniques (8), while useful, require the prac- 
titioner to the model the eigenvectors of the underlying 
population covariance matrix as well; this makes them 
especially sensitive to model mismatch errors that are 
endemic to high-dimensional settings. This motivates 
our development of an sample eigenvalue based estima- 
tor with a computational complexity comparable to that 
of the WK estimators. 



3. ESTIMATING THE NUMBER OF SIGNALS 



Given an observation y = [y(l), . . . , y(N)] and a fam- 
ily of models, or equivalently a parameterized family of 
probability densities f(y\0) indexed by the parameter 
vector 0, we select the model which gives the minimum 
Akaike Information Criterion (AIC) (TO) defined by 



AIC fe = -21og/(y|0) + 2fc 



(8) 



where is the maximum likelihood estimate of 0, and k 
is the number of free parameters in 0. We derive an AIC 
based estimator for the number of signals by exploiting 
the following distributional properties of the moments of 
eigenvalues of the (signal-free) SCM. 

Theorem 1 ( Dumitriu-Edelman ft2]l) Assume R is formed 
from m snapshots modelled as (|7} with k = 0, A = 1 then 
as to, n — > oo and c m = n/m — > c £ (0, oo), then 



£S=i U - n 
Ew?-»(i + c)-(j-i) 



A/"(0,Q) 



where T> denotes convergence in distribution, = 1 (or 
2) when Xj is rea/ (or complex) valued, respectively, and 



Q 



c 2c(c+l) 
2c (c + 1) 2c (2c 2 + 5c + 2) 



Proposition 2 Assume R satisfies the hypotheses of The - 
orem\l\for some X then as to, n — ► oo ana" c m = n/m — > 
c £ (0, oo), f/ze« 



[i„-(l + c)]^AA( (^-ljc,^c 2 



and the test statistic t n is given by 

n 'I Second moment of eigs 



(9) 



Mean sq. of cigs 



Proof. This follows from applying the delta method 
[ 1 1 1 to the results in TheoremQ] □ 

When k > signals are present and assuming k -C 
n, then the distributional properties of the n — k "noise" 
eigenvalues are closely approximated by the distributional 
properties of the eigenvalues given by Theorem[T]of the 
signal-free SCM, i.e., k = 0. Hence, by evaluating the 
statistic in Proposition|2]over a sliding window, and using 
the normal approximation for the statistic from Proposi- 
tion |2] with c w n/m and k + 1 free parameters in the 
AIC formulation in dD results in the estimator: 



fcNEW = argmin 




"TO" 




+ 2(fc+l), 


fceN:0<fc<min(n,rn) 


14 


. n - 















(10a) 



where 



Qk =n 



1 V^n ;2 
\n-k l^i=k+l H) 



m 



1 



n 

TO 



(10b) 



Here /3 = 1 if Xj G R n , and /3 = 2 if Xl G C". 
When the measurement vectors represent quaternion val- 
ued narrowband signals, then we set j3 = 4. Quaternion 
valued vectors arise when the data collected from vector 
sensors is represented using quaternions as in |[T2l . 

4. FUNDAMENTAL LIMIT OF DETECTION 

The following result exposes when the "signal" eigenval- 
ues are asymptotically distinguishable from the "noise" 
eigenvalues. 

Proposition 3 Assume R satisfies the hypotheses of The- 
orem [7] Denote the eigenvalues of R by \\ > A2 > 
. . . > Afe > Afe + i = . . . X n = A = a 2 . Let lj denote the 
j-th largest eigenvalue of R Then as n, to — > 00 with 
c m = n/m — ► c G (0, 00), and j = 1, . . . , k + 1, 



A, 1 + 



A; 



where the convergence is almost surely. 



ifA J>( 7 2 (l + ^) 
if Xj < <7 2 (1 + y/c) 



(ID 



PROOF. This result appears in |fl3l for very general set- 
tings. A matrix theoretic proof for when c < 1 for the 
real case may be found in fl4l and an interacting particle 
system interpretation appears in |[T5l . □ 

Motivated by Proposition [3] we define the effective 
number of signals as 



fceff(R) = # eigs. of R > a- 1 + W — 



(12) 



4.1. Identiflability of closely spaced signals 

Suppose there are two uncorrelated (hence, independent) 
signals so that R s = diag(cr| 1 , <t| 2 ). In (Q~|i, let A = 
[vi v 2 ] . In a sensor array processing application, we think 
of V! = v(#i) and v 2 = v 2 (# 2 ) as encoding the array 
manifold vectors for a source and an interferer with pow- 
ers ct^ and (Tg 2 , located at B\ and # 2 , respectively. The 
covariance matrix given by 



R = (Jgj^VlV^ + <7g 2 V2V 2 + £T 2 I 



(13) 



has the n — 2 smallest eigenvalues A3 = . . . = A„ = a 2 
and the two largest eigenvalues 



Ai 



+ 



2 , (>LlMI 2 +^2lHI 2 ) 
r ' 2 

\/^i 1 hi\\ 2 -^hh2\\ 2 ) 2 +4al 1 ai 2 \( Vl , V2 )\ 2 



(14a) 



Xa = <T 



KlMI 2 +^ 2 IHI 2 ) 
2 



respectively. Applying the result in Proposition [3] allows 
us to express the effective number of signals as 



if a'[l + J- <A 2 

m 



if A 2 < a 2 ( 1 + W— I < Ai 



if A, < cr 2 1 



k e s = < 





V V m J 

(15) 

In the special situation when || Vi || = || v 2 || = || v || and 
°Si = °S2 = °s> we can ( m an asymptotic sense) reli- 
ably detect the presence of both signals from the sample 
eigenvalues alone whenever 



al llvll 2 [ 1 



|(vi,v 2 )| 



> a' 



(16) 



Equation ( TTol l captures the tradeoff between the identifl- 
ability of two closely spaced signals, the dimensionality 
of the system, the number of available snapshots and the 
cosine of the angle between the vectors vi and v 2 . It may 
prove to be a useful heuristic for experimental design. 
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(a) Prob(fc = 2) versus n for fixed n/m. 
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(b) Prob(fc = 1) versus n for fixed n/m. 

Fig. 1. Comparison of the estimators over 20,000 trials. 

5. SIMULATIONS 

Assume the covariance matrix R has n — 2 "noise" eigen- 
values with a 2 = 1, and two "signal" eigenvalues with 
Ax = 10 and A 2 = 3. When m = An samples are avail- 
able, Figure [T(a)| shows that the proposed estimator con- 
sistently detects two signals while the WK MDL estima- 
tor does not. However, when m = n/4, Figure [1(a)] sug- 
gests that neither estimator is able to detect both the sig- 
nals present. A closer examination of the empirical data 
presents a different picture. For the covariance matrix 
considered, when m = n/4, then from (|121l . k e jj = 1. 
Figure [T(b)1 shows that for large n and m = n/4, the new 
estimator consistently estimates one signal, as expected. 
The WK MDL estimator detects no signals. We conjec- 
ture that the new estimator consistently estimates k e f / in 
the n, m — > 00, n/m — > c sense. 



6. CONCLUSIONS 

An estimator for the number of signals in white noise was 
presented that exhibits robustness to high-dimensionality, 
and sample size constraints. The concept of effective 
number of signals described provides insight into the (asymp 
totic) regime in which reliable detection with sample eigen- 
value based methods, including the proposed method, is 
possible. This helps identify scenarios where algorithms 
that exploit any structure in the eigenvectors of the sig- 
nals, such as the MUSIC and the Capon-MVDR algo- 
rithms in sensor array processing, might be better able to 
tease out lower level signals from the background noise. 
It is worth noting that the proposed approach remains rel- 
evant in situations where the eigenvector structure has 
been identified. This is because eigen-inference method- 
ologies are inherently robust to eigenvector modelling er- 
rors that are endemic to high-dimensional settings. Thus 
the practitioner may use the proposed estimator to com- 
plement and "robustify" the inference provided by algo- 
rithms that exploit the eigenvector structure. 
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